Papers with diagnostic tasks probing
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept. |
| Approach: | They propose a method for automatically building probe datasets from expert knowledge sources, allowing systematic control and a comprehensive evaluation. |
| Outcome: | The proposed model is predisposed to recognize certain types of structural linguistic knowledge, but performance degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy. |